The invention involves an image representing device, in particular a microscope, in particular a stereomicroscope, for example an operation microscope, in particular a video stereomicroscope that is linked with an electronic data processing unit and/or a display and a method of superimposing an image from a second image representing device such that the two images conform geometrically.
Such microscopes are used in technology, among other things, e. g. materials technology, material analysis, silicon technology, criminology, etc., in particular, however, also in medicine for diagnosis, serological examinations, in operations, etc.
Detailed examples will be given below of the use of the device in the sphere of operation microscopy. The use in other spheres, however, likewise falls within the sphere of application of the invention.
Operation microscopes are used by the operating surgeon for the optical magnification of the operating area. Operation technology in this regard has made such great progress that magnifications in the range of 50 times and more are no rarity. A magnification sometimes leads to a situation in which the operating surgeon cannot always unambiguously identify the area that he is viewing through the microscope with an actual place on the patient. A helpful superimposition of the image seen microscopically with intensified contours, for example, or other markings is therefore often desirable. For the fulfillment of this desire essentially two processes are known in the prior art:
Edge improvements, image coloring, contrast improvements, etc. are effected by means of electronic image processing. The image data required for this are obtained directly from the image being viewed itself and simply transformed mathematically. If such data are superimposed on the observed image data, no significant problem arises in the process, since these image data are returned to the same image location from which they were obtained.
In other designs beam splitters to which displays are allocated through which the images in the light path can be reflected are linked with the microscope light path, and these images are superimposed on the images actually seen in the eye of the viewer. Such images, e. g. images of the same place but taken at an earlier time, are often difficult to superimpose on the image that is seen, since they may be taken with different magnifications, other microscope settings, etc.
A special sphere for the superimposition of images arises, for example, in the use of computer tomography (CT), magnetic resonance imaging (MRI), or positron emission tomography (PET) in connection with stereo microscopy. Data from CT and MRI are obtained in order to have a sectional image of the area of interest, which in the final analysis after computer processing makes it possible to display a three-dimensional model that is true to life on a computer monitor (stereo screen). In this regard see Chapter 1 (xe2x80x9cInterventional Video Tomography (IVT)xe2x80x9d), pages 1-25, Chapter 2 (xe2x80x9cDigital Substraction Angiography (DSA)xe2x80x9d), pages 27-34, and Chapter 3 (xe2x80x9cTechnical Specificationxe2x80x9d), pages 35-41, written by M. Truppe in the company journal of ARTMA BIOMEDICAL, INC. (Apr. 7, 1994).
U.S. Pat. No. 4,722,056 and CH-A-684291 describe devices that theoretically make superimposition possible. A computer-assisted operation microscope, which was put on the market under the designation MKM, can mark both risk zones and tissue structures and also superimpose three-dimensional diagnostic data that are obtained before the stereoscopic viewing on the image seen in the microscope in order to make forward-looking operating possible. An exact superimposition, so that outline contours from the CT, for example, coincide with the stereoscopically viewed outline contours, however, is not possible with this.
Through three-dimensional images it is possible for the attending physicians to localize the kind and extent of the diseased area better and plan appropriate operation steps better in advance. The three-dimensional image data provided by the computer are now in accordance with the invention supposed to be accessible to an operating surgeon in an improved manner immediately before an operation too, specifically in such a way that these image data are exactly superimposed on the image seen in the operation microscope at the same time and in the right position; this may be also in a masking mode or a transparent mode, which makes it possible for the operating surgeon to see as it were under the surface of the place actually being seen and in this way to make possible improved planning and guidance of the operating instrument. This should result in a higher precision of positioning and shorter operation time than is possible today.
In order for this superimposition to take place optimally the optical parameters of the image seen and the parameters of the (three-dimensional) image to be superimposed must be known, since such a superimposition makes sense only if the image seen through the microscope and the superimposed image data conform geometrically. Geometrical correctness is not satisfactory to date for the known superimpositions.
Overcoming this situation is one of the main objectives of the invention.
The objective is attained, for example, through an adaptive control apparatus according to the present invention;
The first application of an adaptive control apparatus to modify an image to be displayed depending on another displayed image leads to the desired success. In this regard it is first of all immaterial where the actual image superimposition takes place. The following variants are listed as examples:
When one two-dimensional image is displayed exclusively for one of the two eyes of a viewer and another two-dimensional image is displayed for the other viewer eye, the superimposition takes place in the brain of the viewer. It can, however, just as well take place in the computer in order to deliver the superimposed image as an integrated image signal to a monitor or similar device; both signals, however, can also be delivered directly to a monitor or similar device that has, for example, two input channelsxe2x80x94if appropriate, for a right and left image channel of a pair of stereo images. Further a purely optical superimposition is conceivable, e. g. in the intermediate image of an optical light path or something similar.
Various qualitatively different measures are provided for the control apparatus in the framework of the invention to detect and correct the image geometry or imaging parameters.
It must always be ensured that the control apparatus detects primary (operational) imaging parameters (eyeline, thus system alignment or viewing angle, perspective, etc., e. g. microscope alignment) and uses them for adapting the image formation geometry of the second image information. This is implemented, for example, through picking up the direction data and settings of the adjustment means of the first device, e. g. from the stand or through monitoring the microscope position through an independent sensing device, such as an ultrasound or infrared position or coordinate determination device (e. g. PIXSYS(trademark)) and the device for setting the magnification of the microscope.
An improved adaptation results if the secondary imaging parameters too (e. g. field of vision, magnification, depth of focus) are detected and used for adapting the image formation geometry of the second image information. This is implemented, for example, through picking up the actual z-distance (this is the distance from the object being viewed to the objective bearing surface) from the object viewed and a magnification measurement (gamma measurement) in the light path of the first device.
Obviously it is optimal when the tertiary imaging parameters (e. g. aberration of the metric, distortion depending on the z-distance and gamma measurement in the light path of the first device) are detected and used to correct the image formation geometry of the second image information. Such tertiary imaging parameters are preferably carried out through comparative measurements on known objects (e. g. illustrations with a known geometric pattern or three-dimensional objects with known geometric dimensions that can be utilized for calibration purposes or to detect the imaging errors).
The primary and secondary imaging parameters are type-specific parameters that depend on the nature and geometric or optical design of the first device, while tertiary imaging parameters are series-specific. They depend on the quality of the individual optical components and their assembly and on the adjustment of each individual apparatus.
Naturally an adaptive control apparatus is preferred that detects primary, secondary, and tertiary imaging parameters and on their basis modifies the second image data that are to be superimposed and superimposes them. In the process the second and tertiary imaging parameters may be evaluated first, if appropriate, and only then is a conformation of the coordinate systems of the two devices undertaken in which one image is rotated or pivoted by the computer into the other, which is already displayed in a geometrically correct manner, this, naturally, preferably three-dimensionally in each case.
The necessary calibration processes involved may be carried out using the whole adjustment range of the system data of the devices where, for example, a permanent calibration is made during a slow adjustment of all the system data of an apparatus and the various computational formulas that are derived in the process are recorded continuously and stored in a memory, so that later it will be possible to superimpose the appropriate correction value for each individual setting during the operation of the apparatus.
In order to reduce somewhat the quantity of image data that must be processed by the computer and to speed up the processing, a further development of the invention is foreseen in which the necessary correction rule for the type-specific algorithms (optical distortions that can be determined mathematically, etc.) are calculated ahead of time and stored in the device, so that in the actual adaptation an automatic calculation will take place first and only thereafter a measurement with further adaptation results.
To facilitate and accelerate the elimination of the series-specific imaging parameters, it is preferably proposed to subject each device to a calibration process after which the calibration parameters that have been obtained are stored in a memory and are automatically utilized in the adaptation of the second image data. Thus, as soon as images from another image source are superimposed in a particular type of device or piece of equipment and a particular piece of equipment from a particular series, these images will be modified with the stored values in accordance with a calibration carried out before the delivery of the equipment in such a way that the images can be superimposed congruently, to the extent that they come from the same object. In the process it makes sense to carry out the calibration for each individual device. If needed it is even conceivable that a standard component appropriate for the calibration can be used that will be directly available to the local service personnel or even the user, to make recalibration possible from time to time.
The advance calculations and calibrations or storage of correction values that have been described are suitable for equipment that has a limited memory or computing capacity. If there is adequate computing capacity and appropriate computing speed, it is also conceivable to supply equipment that is optimized to a certain degree but without precalculating the type-specific algorithms and to build into the computer an image recognition apparatus to which the precise dimensions of a standard component are known; in certain intervals it calls preferably for a calibration routine in which through video viewing of the standard component alone the deviation from the stored dimensions are detected and from them are determined the correction parameters, which are written to the superimposed images in each case. As a standard component an object is preferred that presents suitable, recognizable structures for the imaging rays of the first and second apparatus.
In calibration it makes sense to proceed as follows. A test object with dimensions that are standardizedxe2x80x94and preferably recognizable for all image producing light paths that are usedxe2x80x94is scanned with its actual dimensions using CAD and stored in a memory by a computer as a voxel volume with external dimensions conforming to reality. Subsequently this test object is placed in front of the light path of the first apparatus, which has a stereoscopic video recording apparatus (e. g. one CCD per light path of a stereomicroscope) and through this observes from several perspectives. Each of the views that are allocated to the individual perspectives is taken and converted into a voxel volume by means of a computer program, which [voxel volume] thus corresponds to reality, since it was modified (e. g. distorted) by the imaging parameters. The voxel volume so obtained is then comparedxe2x80x94e. g. by voxelsxe2x80x94with the voxel volume that was originally input into the computer. The differences that result from this are recognized as a computational formula and stored in the memory. As soon as another object is placed in the light path of a microscope with the same setting, it can be modified by means of the computational formula in such a way that it corresponds to its true proportions inside the computer or memory and can also be displayed as such.
In most cases the process just described with the recalculation using the voxel volume is carried out only because resort is made to the voxel volume when one wants to look under the surface of the object being viewed in the xe2x80x9cmasking mode,xe2x80x9d as is made possible in the correct position through the invention. Then, namely, the voxel volume beneath the place viewed by the microscope is taken away by layers as necessary, so that one can look into the depth by layers. Naturally the 3-D display of all or a part of the voxel volume is also possible. For this a weighting that can be set optically or using software of the image brightness of the microscope image or the superimposed image is possible. For pure calibration the analysis of the pure projection of the standard component being viewed is sufficient through which its recognized edge lines or their corner points are stored as vectors and can be processed further. On the other hand, and this is an important point of one of the embodiments of the invention, other xe2x80x9ccorrectxe2x80x9d data, e. g. from a CT, etc., that are available through the computer can be revised with these computational formulas in order to correspond geometrically correctly to distorted microscopic image data, for example.
Obviously it is often important to take into account the actual position of an object in space in this regard, especially in stereotactic operations or the like. For this a position recognition device is foreseen in connection with the invention that measures both the object (test object) and the position of the microscope in space. As an example, the system of the company Pixsys is suitable as a position recognition device in which IR transmitters or IR reflectors are mounted on the object and on the camera at known intervals and their position in space is determined by the system and recorded for further processing. The invention is not confined to this system, however. For most embodiments it is important, in any case, for sensors or signal markers to be arranged on the microscope that work together with signal markers or sensors located in space to define the position of the microscope in space. For example, instead of the sensor-signal marker arrangement a pure video image processing could be used in which one or two fixed video cameras viewed the microscope and after advance calibration calculated the spatial position of the microscope by means of 3-D image processing. Alternatively, of course, the MKM system in the stand that was already mentioned can also be used in which sensors in the stand or its joints determine the position of the microscope.
It does not matter which of the images finally displayed to the user is used for the corrections. It is just as conceivable to adapt the image of the optical light path to that of the nonoptical light path as vice versa, or also to create a xe2x80x9cstandard formatxe2x80x9d to which both the optical and nonoptical images may be adapted. Such a xe2x80x9cstandard formatxe2x80x9d could also be selected in such a way that it is closest to reality, or that it is portrayed virtually as a viewer of the corresponding object with average visual acuity would actually see the object. In the case of viewing through a microscope, the average viewer naturally would be reduced virtually in proportion to the magnification of the microscope.
As a rule imaging through CT is fairly close to reality. If, however, imaging distortions or something similar are present here too, these can also be taken into account through being compared to a calibrated object and displayed as a computational formula in order to superimpose these deviations on the image data from the first device too. Through such a xe2x80x9ccrosswisexe2x80x9d superimposition of correction data, identical images are generated in both the first and second apparatus.
Of course it is also conceivable to correct the image data in each case after calibration in such a way that they correspond to the magnification of the object viewed that is only mathematically correct, whereupon the image data are also congruent and can be superimposed.
The adaptation of the image from the light ray path can take place in the first devices in which the light ray path is not sent directly to the viewer""s eye, as is the case, for example, in videomicroscopes, where the viewer looks at a screen and not at the actual light ray path.
The superimposition of the images may take place either directly in the ocular light path or through the use of software in the computer, which displays the superimposed (combined) total image on a monitor. What is important in the final analysis is the fact that what the viewer sees is a superimposition that is in the correct scale and true to reality or appearance, as though both of the originally independent images had been viewed through one and the same optics with the same adjustment.
The approach to the problem in which it is required, for example, to eliminate the errors of the microscope""s light path and thus to generate an image that is true to life, instead of transferring the errors to the other image, generally does not achieve a great deal for an operating surgeon, since in any event he does not work in a true life environment because of the magnification. His direct point of reference is the operating instrument, which he perceives in the field of vision of the microscope and moves by hand. An adjustment of the microscope light path or its optical elements leads automatically in accordance with a special embodiment of the invention to an adaptation of the reflected (superimposed) image, so that the operating surgeon can continue to work without interruption.
The term xe2x80x9cdisplayxe2x80x9d as used in the invention encompasses, for example, screens, cathode ray tubes, etc. on which the images appear for the user. Such displays may be located either outside the microscope, e. g. as a computer monitor, or also as small displays that are built into an ocular light path in such a way that a user obtains not only an optical perception from the optical light path but also simultaneously superimposed an optical perception from the display. The purely electronic superimposition of images and their presentation on a display outside the microscope are encompassed by the invention, the ocular light path in one such case being reflected via a beam splitter into at least one image reception apparatus.
The term xe2x80x9cvideomicroscopexe2x80x9d as used in the invention encompasses microscopes with at least one light optical ray path and at least one image reception apparatus for receiving and presenting the image seen via the light path on a display. Recently videostereomicroscopes in which two parallel light paths are provided and a 3-D image can be presented on the display have become a very widely used kind of videomicroscope. The invention includes, however, all other microscopes and endoscopes that have the devices described above and if necessary do without a stereoscopic light path.
Above all in operation microscopes and especially during an operation a lot of information comes in that may be of great significance for the surgeon. This includes, for example, information about the patient or his state of health such as pulse, blood pressure, oxygen content of the blood, etc., but also in addition to the images to be superimposed that have been mentioned, e. g. information concerning certain parameters of the microscope, information on the position of the operation field being observed, as well as control data that is transmitted arbitrarily via control elements such as the computer mouse or foot switch to the data processing or to control elements for the microscope in order to control it as needed, e. g. for focusing, etc.
Within the scope of the invention data can also be superimposed optically, through the electronic input of these data into the second image that is shown, or through the use of an additional display that is blended into the microscope light path. The superimposition of several different data items, e. g. also written data, is possible.
In the commonly-owned International Publications WO 9527917 and WO 9527918 claiming priority of Swiss patent application CH-1091/94-3 a device is described that makes it possible to carry out the superimpositions and data adaptation as fast as possible or in real time. A combination of the teaching of this invention with the teaching of the application mentioned thus brings further advantages. To this extent the content of the patent application mentioned is considered to lie within the framework of this disclosure.
CH-A-684291, which has been mentioned, describes a process for detecting the position of an image plane. This process could also find application in the framework of the invention in that one refrains preferably from a triangulation measurement, and the distance as well as the focus are determined directly through image evaluation (e. g. marginal definition evaluation).
To this extent and for practical implementation by an person skilled in the art the content of the patent application mentioned and CH-A-684291 are also regarded to lie within the framework of this disclosure.